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Jaynes' maximum entropy (MaxEnt) principle was recently used to give a conditional, local deriva- 
tion of the "maximum entropy production" (MEP) principle, which states that a flow system with 
fixed flow(s) or gradient(s) will converge to a steady state of maximum production of thermody- 
namic entropy (R.K. Niven, Phys. Rev. E, in press). The analysis provides a steady state analog of 
the MaxEnt formulation of equilibrium thermodynamics, applicable to many complex flow systems 
at steady state. The present study examines the classification of physical systems, with emphasis 
on the choice of constraints in MaxEnt. The discussion clarifies the distinction between equilibrium, 
fiuid flow, source/sink, flow/reactive and other systems, leading into an appraisal of the application 
of MaxEnt to steady state flow and reactive systems. 
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1. INTRODUCTION 

Half a century ago, Jaynes established the maximum 
entropy (MaxEnt) principle as a method of inference for 
the solution of indeterminate problems of all kinds [1- 
3], based on the relative entropy function (the negative 
KuUback-Leiblcr [4] function): 

s 

where pi is the probability of the ith distinguishable cat- 
egory or choice within a system, from s such categories, 
and Qi is the source or "prior" probability of category 
i. Maximization of (1), subject to the constraints on a 
system, yields its "least informative" or "maximally non- 
committal" probability distribution. Using the generic 
"Jaynes relations" , this can then be used to infer the 
macroscopic properties of the system. Jaynes' method 
has been applied to the analysis of a vast number of phe- 
nomena throughout most fields of human study [e.g. 3, 5] , 
and can be regarded as one of the most important dis- 
coveries of science. 

Three decades ago, a new principle was proposed by 
Paltridge for the analysis of flow systems, the maximum 
entropy production (MEP) principle [6, 7]. This can be 
specified as "a /low system with many degrees of freedom, 
subject to fixed flows or gradients, will tend towards a 
steady state position of maximum production of thermo- 
dynamic entropy^^ . Since its proposition, the MEP prin- 
ciple has been applied successfully to predict the steady 
states of a wide range of systems, including the Earth's 
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climate system (oceans and atmosphere) [e.g. 6-8]; ther- 
mal (Benard) convection [9]; material flow through eco- 
logical systems [10] and biochemical processes [11, 12]. 
Lying outside present-day thermodynamics, the MEP 
principle appears to provide a unifying principle for the 
analysis of flow systems of all kinds. 

Until recently, the theoretical basis of the MEP princi- 
ple was unclear, limiting its acceptance. Philosophically, 
however, the MEP principle concerns reproducible be- 
havior of a system, and therefore must be consistent with 
- and hence derivable from - Jaynes' MaxEnt method. 
The first approach in this direction was undertaken by 
Dewar [13, 14], who examined a time- variant nonequilib- 
rium system, using a MaxEnt analysis based on the prob- 
abilities of paths in phase space. More recently, the au- 
thor [15] has given a local, conditional MaxEnt derivation 
of MEP, using the probabilities of instantaneous fluxes 
through each infinitesimal element of a flow system. 

The aim of this study is to explore the conceptual 
framework of the latter analysis in more detail. Firstly, 
various types of physical systems are classified, to more 
fully understand how Jaynes' MaxEnt method can be 
applied, especially the handling of various types of con- 
straints. Several formulations of the MaxEnt method are 
then discussed, with application to equilibrium, flow and 
source/sink systems. The analysis leads into a discus- 
sion of Lagrangian multiplier constraints and the role of 
the potential function (generalized free energy concept) 
in MaxEnt analysis. 



2. TYPES OF SYSTEMS 

Consider a physical system composed of discrete enti- 
ties, each of which may adopt particular values of one or 
more physical quantities. In general, the system will be 
free to roam throughout its available "category space" . 
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In particular cases, however, the system will be con- 
strained by mean values of particular physical quantities, 
whereupon the "least informative" or "most probable" 
state of the system can be inferred by Jaynes' MaxEnt 
method. 

With some thought, many physical systems can be 
classified into the following scheme, as illustrated in Fig- 
ure 1: 

• Isolated systems (Fig. la), consisting of a set of entities 

enclosed by some impermeable barrier, constrained by 
mean contents {fr) of various physical parameters fr, 
r = 1, ...,R. If the content constraints are constant, 
such systems will converge towards an equilibrium po- 
sition, which - as is well known - can be inferred by 
MaxEnt [1]. 

• Flow systems (Fig. lb), defined by a control volume, a 
geometric region through which fluid(s) may flow (the 
Eulerian description), constrained by mean flow rates 
{3^r) of various physical quantities fr through the sys- 
tem, r = 1, R. If the flow rates are constant, such 
systems will converge towards a steady state position, 
which - in principle - can also be inferred by MaxEnt 
[15]. 

• Source/sink systems (Fig. Ic), consisting of a set of enti- 
ties enclosed by some impermeable barrier, constrained 
by mean rates of change (fr) of various physical pa- 
rameters /,. , r = l,...,i?. Such systems encompass 
chemical reaction kinetics, radioactive and biological 
growth/decay processes. If the rate constraints are 
constant, such systems will converge towards a con- 
stant growth position, which - arguably - can also be 
inferred by MaxEnt. 

• Flow and reactive systems (Fig. Id), again defined by a 
control volume, but in this case with different incoming 
flow rates {3^r)in, outgoing flow rates {'Jr)out and rates 
of production {fr) of various physical quantities. If the 
flows and rate of production of each quantity are in 
balance, such that a system attains a steady state flow, 
it should be possible to infer this position by MaxEnt 
analysis. 

• Transient systems, akin to those shown in Fig. Id, but 
in general with time- varying incoming and/or outgoing 
flow rates and/or rates of production within the sys- 
tem. In general, such systems may not be amenable 
to analysis by MaxEnt, but in some special instances, 
they may be so amenable. Several such systems are 
examined by Dewar [13, 14] by a path MaxEnt anal- 
ysis, and by Grandy [16] using a quantum mechanics 
formulation. 

Note that we here avoid the standard epithet ^^non- 
equilibrium system", which severely lacks precision; in- 
deed, it is about as useful as the colour description "non- 
blue". 

How can the MaxEnt method be applied to each above 
case? After a moment's reflection, it will be apparent 

that in isolated systems (Fig. la), the physical quantities 
fr within the mean constraints {fr) are (usually) taken 



as extensive variables. By the zeroth law of thermody- 
namics, such properties should be uniformly distributed 
throughout the system at equilibrium. When applying 
MaxEnt to an isolated system, we are therefore justi- 
fied - in most circumstances - to analyse the system on 
a whole-of-system basis. This approach does not, how- 
ever, apply in general, especially to the analysis of flow, 
source/sink and flow/reactive systems. In flow systems 
(Fig. lb), for example, although the bulk flows into and 
out of the control volume may be specified around the 
control surface, many flow patterns could exist which 
are consistent with such boundary conditions. For max- 
imum rigour, it is therefore necessary - consistent with 
the formulation of other laws of fluid mechanics - to ap- 
ply MaxEnt to each infinitesimal element of the system. 
The analysis must therefore be conducted usings fluxes 
rather than flow rates^. Similar considerations apply to 
source/sink, flow/reactive and transient systems, which 
also force the user to adopt a local rather than whole-of- 
system formulation of MaxEnt. 



3. MAXENT ANALYSES 

We now turn to the mathematical treatment of each 
case. In an equilibrium, system (Fig. la), we consider 
the joint probability Pt that an entity adopts particular 
values fri^ of the physical parameters fr, r = {1,...,R}. 
We therefore adopt the relative entropy: 

^e9 = -^ftln- (2) 
t * 

where i, is the index of categories for the rth constraint, 
t = {i\, ...in} and is the joint prior probability. In ther- 
modynamics, the prior probabilities are typically handled 
in terms of the degeneracy gi^ = qi^Gr of each category 
V G I, where Gr = Ylii 9i,-- EIQ- (2) is subject to the 
natural and moment constraints: 

X]ft = l, ^Pvfn^ = {fr), r = l,...,R, (3) 

where {fr) is the mathematical expectation of fri^. Typ- 
ical thermodynamic constraints include the mean inter- 
nal energy {U), mean volume (V) and mean numbers 
of moles (ric) of particles of each type c. Maximiza- 
tion of (2) subject to (3) by Lagrange's method gives 
the inferred distribution, "Jaynes' relations" , generalized 
Clausius equality and potential function of the system, 
as listed in Table 1. We can further identify the dimen- 
sionless entropy and multipliers as functions of (histori- 
cally) known variables; e.g. the thermodynamic entropy 



^ In engineering analysis, fluxes are vectors or tensors, expressed 
in SI units of quantities s~^, whereas flow rates are scalars, 

expressed in quantities s~^. 
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FIG. 1: Types of systems: (a) quantity-constrained (equilibrium) systems; (b) flow-constrained (steady state) systems; (c) 
rate-constrained (source-sink) systems; and (d) flow and reactive systems. 



\s S ~ k^]* , where k is the Bohzmann constant, whilst 
the multipHers for the constraints (U), {V) and {ric) are 
Xu = l/kT, Xv = P/kT and Ac = -fJ-c/kT, where T 
is the absolute temperature, P is the absolute pressure 
and /ic is the chemical potential of the cth constituent. 
In addition, the potential (f)eq equates to the Planck po- 
tential of the system, equivalent to the applicable free 
energy divided by kT [15]. With these identifications, 
the relations in the left column of Table 1 provide the 
foundations of equilibrium thermodynamics [e.g. 17]. 

We now consider steady state systems, as illustrated in 
Fig. lb, for which - as noted earlier - it is necessary to 
apply MaxEnt to each infinitesimal element of the sys- 
tem. We thus consider the joint probability tt i that a 
volume element experiences instantaneous values j^^ of 
the fluxes of various quantities fr through the element. 
We therefore maximize the relative entropy: 

ij.t = -V vri In— (4) 
J 71 

where I — {ii, ...i/?} and 71 is the joint prior probability. 

can be termed the flux entropy, since it expresses the 
spread of the distribution of instantaneous local fluxes; it 
is fundamentally difi'erent to the thermodynamic entropy 
S = kSjlg. Eq. (4) is subject to the natural and moment 
constraints: 

^TTl^l, ^T^I jrt^ (Jr), r=l,...,R, (5) 

I I 

Typical constraints of interest include the mean heat flux 
(Jq), fluid flux (j f), mass fluxes (j^) of each species c, 
momentum flux (stress tensor) (r) and/or charge flux 
(jz)- Application of MaxEnt then yields the inferred dis- 
tribution and set of relations for a flow system at steady 



state, as listed in Table 1, where (^r are the new (vec- 
tor or tensor) Lagrangian multipliers, is the Massieu 
function, Z is the partition function and "•" represents 
the vector or tensor scalar product (as circumstances re- 
quire) . 

The flow system parameters listed in Table 1 can be 
further identifled as functions of known parameters. An 
important quantity of "non-equilibrium thermodynam- 
ics" is the entropy production, given for each infinitesi- 
mal element (per unit volume) by [18-20]: 

^ = J2'' ,^Yr- (Jr) (6) 
^ — ^r— 1 

where Yr is the extensive variable associated with quan- 
tity fr, whilst V is the Cartesian gradient operator. 
Comparing (6) with the flux entropy at steady state, 
^*st = Co + • (jr), wc scc that both quantities 

contain a sum of scalar products of fiuxes with other 
quantities. By monotonicity arguments for each rth pair 
of fluxes and gradients in (6), making use of the fact that 
the fluxes are linearly independent, we can therefore iden- 
tify the Lagrangian multipliers as: 

ev 

C = -J^^^r (7) 

where 9 and V are characteristic time and volume scales 
for the system. With these identifications, the relations 
on the right in Table 1 provide the foundation for the 
MaxEnt analysis of flow systems at steady state. 

Source-sink systems can be analysed by MaxEnt in a 
similar manner. Adopting the entropy (4), now based 
on the joint probability ttj of instantaneous local rates 

of production of quantities d, subject to constraints 
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■Qeq = -EP'ln — 

t ?> 
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I 71 


Inferred distribution 


R 

Pt = 2~^g.exp(- E Ar/rir) 
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TTl = exp(- E -Jrir) 
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Z = = Eft exp(- E Ar/r-v) 
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Jaynes relations 


R 
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d^Xo _ d(fr) 
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9'CO _ d(jr) 






Ft/" Ft/" Ft/" 


Clausius 


R 

dSjlq = E Ar^Qr- 
r=l 


R 

d^tt = E Ct- • ^Qr 

r=l 


Potential function 


R R 
#eq = -dAo = E XrSWr + E dXr(fr) 
r—l r=l 


R R 

dcpst = -dCo = E Cr • SWr + E <^Cr • (jr) 
r—l r = l 




= -dS}:^+df:Xr{fr} 
r=l 


R. 

= -dS)*t + d E Cr • (jr) 
r=l 



TABLE I: Assumptions, entropy function, inferred probability distribution and Jaynes relations, for (a) equilibrium and (b) 
steady state flow systems (B=background information, 5Qr = rth generalized heat, SWr = rth generalized work, Sqr = rth 
flux of generalized heat, 6wr = rth flux of generalized work). 



^ J TTi =1 and ^ J TTi^Cd = (Cd)) one obtains relations 
similar to those listed in Table 1 for flow systems, but ex- 
pressed in terms of {^d) rather than (jV)- For chemical 
reaction systems, the Lagrangian multipliers conjugate 

to (^d) can be identified as Q = {6V/k){Ad/T), where 
Ad is the chemical affinity of reaction d. The local en- 
tropy production a = — "^d^dAd/T again emerges as an 
important quantity in this system. 

For the MaxEnt analysis of simultaneously flow and 
reactive systems portrayed in Fig. Id, at steady state, it 
is necessary to adopt composite constraints, given by one 
side of the balance equation for a conserved quantity: 

(3'r)m + ifr) = {^r)out (8) 

This new application of MaxEnt warrants further de- 
tailed examination. 



4. MULTIPLIER CONSTRAINTS AND THE 
MEP PRINCIPLE 

A final point is to consider the alternative formulation 
of each of the systems discussed in §2-3, in which the 



system is constrained - not by mean values of physical 
variables - but by their conjugate Lagrangian multipliers. 
Two examples are shown in Figure 2: 

• Open equilibrium systems (Fig. 2a), in which the equi- 
librium position is imposed by a surrounding bath of 
fixed intensive variables; and 

• Gradient- constrained flow systems i¥ig. 2b), in which a 

steady state fiow system is produced by the imposition 
of fixed gradients of intensive variables. 

Irrespective of whether a set of physical quantities or 
their conjugate multipliers are adopted as constraints, 
the mathematical method for the analysis of each pair of 
systems is the same. This point was in fact appreciated 
by Jaynes [1], but his discussion is rather oblique. In 
addition to those shown in Figure 2, other multiplier- 
constrained systems are possible. Hybrid equilibrium- 
steady state system,s must also be considered, by the im- 
position of fixed intensive variables of certain quantities, 
and gradients of other quantities; indeed, all combustion 
processes in the Earth's atmosphere may be conceptual- 
ized in this manner. 

Since the work of Gibbs [21], it has been standard 

practice in thermodynamics to use the free energy F as 
the criterion for analysis of open equilibrium systems, 
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Const, gradients =KVA^} 



FIG. 2: Types of systems with multiplier "constraints": (a) intensive variable-constrained (open equilibrium) systems; and (b) 
gradient-constrained (steady state) systems. 



wherein F attains a minimum at equilibrium. From a 
Jaynesian perspective, the free energy divided by kT is 
equivalent to the negative Massieu function or potential 
function of a system: 




= —d^lq — df)Rou 

where ?)rou is the entropy of the rest of the universe. 
The potential (t>eq embodies the second law of thermo- 
dynamics, in the sense that spontaneous changes must 
take place by an interplay between changes in entropy 
within the system, diDgg, and changes in entropy outside 
the system, dSj*jiQu, such that d(j)eq < [22, 23]. 

In a similar vein, it is possible to adopt the poten- 
tial function (j)st as the criterion for analysis of gradient- 
controlled flow systems. By a flow analog of the second 
law, (f)st will also attain a minimum at steady state. From 
its definition in Table 1, (6), (7) and the identification 
^0 = ~4'sti the potential can be written: 

6V ~ 

d(l>st = -dS)l^ - —da (10) 

Conditional on the assumption that d^*^ > 0, we see 
that the minimum in will correspond to a state of 
maximum entropy production a. The analysis therefore 
provides a local derivation of the MEP principle, condi- 



tional on a single assumption concerning the behavior of 
the flux entropy Sj*^. 



5. CONCLUSIONS 

This study examines several kinds of physical system, 
including equilibrium, steady state flow, source/sink and 
flow/reactive systems. In these cases, it is shown that the 
"stationary" or "constant" position of the system can be 
inferred by Jaynes' MaxEnt, with the appropriate choice 
of relative entropy function and constraints. For equilib- 
rium, flow and source/sink systems, constrained respec- 
tively by extensive variable contents, fluxes or rates of 
production, the Lagrangian multipliers can be identified 
respectively as functions of the intensive variables, gradi- 
ents in the extensive variables or chemical affinity terms. 
In all cases, changes in the potential function (negative 
Massieu function) can be used as a criterion for station- 
arity, expressing the interplay between an increase in en- 
tropy (however defined) of a system, and an increase in 
entropy outside the system. For flow and source/sink sys- 
tems, the minimum potential corresponds (conditionally) 
to a state of maximum entropy production. 
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